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AccountabiLities 


Objective: 

Increasing self-service 
opportunities while reducing the 
number of calls that go to an agent 
for Driver and Vehicle questions 


Limited Scope: 5 webpages 

e Register Vehicle Permit Licence 
Plate and Sticker 

e Buy or Sell Used Vehicle Ontario 

e Renew Driver's Licence 

* Driver's Licence 

e Payment Help: Driver & vehicle 
online services 


DAILY - All chats and feedback are reviewed daily by the dialogue team 
= Chatbot Specialist does daily audit of chats - updates flows and 
intents, updates Visio - full day 
= Chatbot and Content roles meet for 45 -60 min to review the chat 
conversations and feedback: 
* create notes in Kanban for opportunity, review and assignment 
e Update flows and intents 
WEEKLY 
= SWAT has 2 @ 30 min meetings ensure alignment of information as 
well as sharing common barriers and potential fixes to connect with 
the responsible areas. 
AS NEEDED 
= Outreach to SWAT partners through Kanban or dedicated meeting 
dependant on item and urgency (60 min monthly) 
= Issues creating new web content 
+ New content errors - Connect with MGCS web 
+ MGCS web - form that requires full solution and content writing 
e ServiceOntario online tools - sharing with DAB for improvement 
* MTO - work with SME to determine page owner, policy, web, Drive 
Test or work through MGCS - some more successful 
= New Content from Web 
e translated and approved - easily updated 
* COVID and resumption of business - changes take time 
* New product - if in scope create new flows taking time 
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Transition Steps 


Content Information and transition 
1. Review Scope project scope 
2. Content update 
a. Visio process 
b. Who to involve 
c. Translation 
3. Continuous Improvement update 
a. What to look for 
b. Adhoc process 
c. Who to involve 
4. Kanban 
a. Use and tracking 
5. Measurement 
a. What we are doing internally 
6. Content Training - ODS 
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IBM Watson training 


1. Review Scope project scope 

2. 1 hour with Olive 

3. Homework — IBM Training 

4. Follow — up session with Olive 

5. More Homework — build a mini bot 
6. Follow — up session with Olive 

7. Shadow with Charles 

8. Do content update 

Up Next 


e One on One session with Olive 
e Planning future Chatbot learning 
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What have we done - IBM Watson training 


e Nov5- 1 hour with Olive Homework: e Nov 26 - Session with Olive 

Oo DEMO: https://watson-conversation- O Shared Q and A to build into your 
duo-dev.ng.bluemix.net/ bot, and the general guidelines doc 

© BUILD: for building a bot. 
https://www.ibm.com/cloud/garage/ © Please take some time to play in 
dte/producttour/create-chatbot- the UAT bot, see what you can get 
plan-trip to work and reach out to me if you 

© IBM Documentation: get stuck. 
https://cloud.ibm.com/docs/services Oo Next session week of December 
/assistant?topic=assistant-getting- 13th depending on comfort levels to 
started proceed. 


e Shadow with Charles 


O Assigned to look at new flow with 
Charles 
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What have we done - Content 


Content 
1. Orientation In August 
2. Transition presentation 
3. Attending daily meetings, SWATs, and Web 
4. Invited to Al Community of Practice 
5. Some introductions complete 
6. Content update 
a. Visio process 
3. Continuous Improvement — cc's on all emails 
4. Kanban 
a. Assigned as primary, reviewed use and 
Brad has added information 
5. Measurement 


a. Reviewed Dashboard and attended 
meeting with ServiceOntario 


Content Courses available - ODS 


TBD 


Continuous Improvement 
e Shared Kanban 
e Discussed 
e Voice of the customer 
e Alignment 
Up Next 
e Create outline of Design Supports 
e Brad to lead and plan daily meetings 
e Schedule work sessions with Charles 
e Kim to provide support and coaching 
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= 
U pdati ng Co nte nt RFQ 2020-16 - Extension Letter_signed.pdf 


+ Process - VISIO Kanban - All chats and feedback are reviewed daily by the dialogue 
team 


Content Objective: = Chatbot Dialogue Role does daily audit of chats - updates flows and intents, 


Chatbot answers Driver and Vehicle updates Visio - full day 
questions and is accessed from = Chatbot Dialogue Team meet for 45 -60 min to review the chat 
Ontario.ca. The content is expected to conversations and feedback: 


* create notes in Kanban for opportunity, review and assignment 
= SWAT roles meet for 30 min to review the chat conversations and feedback: 
* create notes in Kanban for opportunity, review and assignment 


align with the web and all channels 
(application, contact centre, media, other 
web, in-person, IVR) and Ministries and 
parts there of (MGCS, ServiceOntario, + Who to involve - CMI Change Management will now transition to the role of the 


MTO, DriveTest)and Vehicle questions dialogue team, managing the daily role of the chatbot with two people to back each 
other update 
= Chatbot Dialogue Team does daily audit of chats - updates flows and 


Limited Scope: 5 webpages intents, updates Visio - full day 
* Register Vehicle Permit Licence Plate = Communications roles meet for sharing of upcoming changes and 
i alignment 

and Sticker = Support roles - meet to discuss trends and answers and to ensure 
e Buy or Sell Used Vehicle Ontario alignment 
* Renew Driver's Licence + Translation - Translation is contracted to CCMB for chatbot translation - 
¢ Driver's Licence ServiceOntario content planning will need to consider this before expiry - March 31, 
e Payment Help: Driver & vehicle 2022 z 

5 = OPS contact is LilliaJouanne@ontario.ca who is a corporate translation 
online services advisor at Business Development and Services, OSS 
= Send English translations in and Excel document for a word count 
and she will send on to the vendor 6 
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Continuous Improvement 


Alignment 
Objective: e We know there is a multi-channel journey for people seeking to 
: : A confirm what they need to do 
Continuous improvement 1 e When people call they will typically have the webpage open or a letter in 
primarily chatbot improvement, hand 
focused on alignment of messaging * Approach: 
across all channels and between e Travel their journey to see what they see find: 
Ministri ° Different language, information, level of detail across web 
AVENUES, (MGCS & MTO), application, social media, IVR, DriveTest, what 
the agent says 
Continuous Improvement e Take findings to Kanban, meetings, and complete and MGCS form for 
+ What to look for WES 
e Follow-up - various level of success in changes 
-  Adhoc process 
«e Who to involve * "Voice of the Customer” is here. 


e Service Issues - Top issues for service, online and payment issues are 
the top issues, followed by Delivery of cards 
* Information Issues - DriveTest, G licence instructions and 
appointments, Over 80 are confused, RIN definition, expectations that 
everything should be online 
e New projects check in -When D & V is in the news or changes made 
to a program the chatbot hears it right away 
* Some project will reach out seeking information 
e Chatbot chats will identify the impact of a change. what is 
confusing, or wrong and share back 
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Tracking and Measurement for Improvement 


Tracking 

Objective: * Kanban - Is an asynchronous way of working and tracking. 
e We track what needs attention, who can help, what this issue is and the 

To keep focused on the goal of priority - by assigning people it loops in people right away. 
improving the customer experience + There are categories for new, work in progress, monitoring, backlog, 
and improvements to the business improvements, future scope, and closed. 
by identifying issues, obstacles and > OWNErSMIp ls With Cll tis, and EEMB 

Measurement 
successes. 


* Dashboard - Is a result of best practices, what we can actually measure, 
what is of interest based on consultation with areas involved. 
Tracking e Measures come from two systems ECCS and IBM Watson 
s S p e Three areas measured - Effectiveness, Adoption, Satisfaction 
Kanban - Use and tracking e ITS is the creator and current owner of the product 
* Operational measures - Specific analysis done at the dialogue level 


Measurement Generally 
. What is done internally e Why are chats unsuccessful? 
+ Why people say no? 
Dashboard * Coverage, Chats/Chats/Calls - evolving 
z KPIs * Chat Feedback 


When looking in detail at every chat (this may not be sustainable) 
e Manual tracking weekly - this is flexible 

* Track specific items online issues, delivery 

« Sharing raw data 
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SWAT Team and Partner Involvement 


Collaboration across Channels, Ministries, Divisions, Branches and Units is essential and working well! 
Two meetings (30 min) weekly with SWAT to ensure alignment of information across the organization 


One meeting (30 min) weekly with Web Coordinators & Social Media to discuss webpage impacts and changes 
As needed asynchronous participation through the Virtual Kanban contact — Teams Daily Planner 


Rarely needed approvals process for content changes 


SWAT 

ServiceOntario 

CMI - Project- Content & Translation — ** Brad Flindall 

CMI - Project —Dialogue Specialist- **TBD 

*CC- Linda Dowling & Sally Campos-Garcia 

CC **CMI — Sarah Loveridge, (Tricia Lewis) 

CC ADMO - Katherine Money 

Digital Adoption Branch — Richard Steeden, Mirko Jokanovic (Alykhan 
Visram) 

MTO 

MTO — Main contact — Nora Challis (Laura Dingwall) 

MTO - Comms Web Co-Ordinator — Andrew Nau & Andrew Miller 
Social Media — Stef Di Risio 

MGCS 

MGCS - Comms -Web Co-Ordinator — Derek Kinley, David Ingraham 
Social Media — Maya Guerrero 

Management - Kristen Polito, Nik Ramkumar 


Supporting Contacts 

ITS 

OPSSC — Bot Creator and Lead - *** Olive Kilpatrick 

OPSSC — Dialogue Specialist - *** Charles Lavigne 

MGCS 

CCMB - Project — Content & Translation — ***Kim Monastyrskyj 
CCMB - Approval process & Content — ***Lyn Lim 

MTO 

Policy Vehicle — content approvals and questions — Arthur Legaro 
Policy Driver — content approvals and questions — Andrew O’Connor 
Drive Test — Gail MacNeil 


ServiceOntario Partner Services — Sarah Palmer 

MGCS Comms - Ashleigh Cocuzzi 

*CC - Customer Care 

**CMI Change Management and Implementation 

***Support Project Dialogue Team to be replaced with Program Area 
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Operational Process for D &V Chatbot 
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[oome DEV Chatbot Process 


Process to review and support quick decisions to make changes to the chatbot content and /or webpage content. 


lor content Review, Updates 
land sot Training 


Branch 


Date: 2021-08-08 
[Prepared by: cms 
[author- kim monastyrskyj 


Reads intonation for 
wrong answer or 

broken paths, copies 

information for 


review iater 


opsscwil capture | 
information forthe | 
next step triage in 
Teams log 


CCMB. OPSSS(ITSD), MGCS Comms, MTO Comms & Policy, ServiceOntario Retail, ServiceOntario Digital Adoption 


Content Teeabisck comes 

to coma from 

* User interactions with 
chatbot, 

e user feedback, 

* sovo/onnequest 

+ SWAT team with new 
information 


«Standard messaging from Webchat 

Framework 

Source is first approved content 
ram web, News, Socal Media 

+ New content requires approval from 


| content 


2 Upowtekaroan wên 
ferunt 


ORE! > RANo ES 


Chatbot Dialogue Team (COT) OPSSC/CC-S0/CCMB 
Dedicated Resources — full time 


© OPSSCD— Bot Technical Specialist 1, 2, 3, 4,5, 6, 
© program Area — Dialogue Business Specialist 3, 5, 
© Asrequired-CCMB- Business Specialist 3, 5, 6,7, 8, 9.$, 82.4, 10, 33, 12 


3 
1,824, 20, 11,12 


Supervisors 
© opssco—Team Lead-6,6.1, 7, 8 


Daily 45 min chat Review Meeting 9:15am (#4) 
‘Tues, Thursday SWAT Meeting 15-30 min @ 230~3:Dopm (#5) 
Webpage Touchpoint Meeting 30 min @ 2:00 pm (#5) 


‘sup porting Channel Communications Leads 
Serviceontario 

‘© metai -Change Management and implementation — 6, 6.1, 7, 8 

© customer care ADM office -6, 6.1, 7, 8 

‘© Customer Care Change Management implementation (CM) - 6, 6.1, 7,8 
‘© Digital Adoption Branch -6, 5.3, 7, 8 


Customer Care Contact Centre -6, 61,7, 8 
Program Area - Business relationship 6, 6.1, 


18, 92-2, 92.3, 9.28, 825 
MTO - SME & Comms Web Co-Ordinator 6, 6-3, 7,8, 92.2, 9.2.3, 9.24, 825 


Participating in SWAT roles to ensure alignment of information across 
channels 


Kanban — 103 Issues closed to date 


Processes have been designed 
with input from all partes. 
Objectiveisto mate quick | JS, 


‘meaningful changes with the 
best use of peoples time. 


bose sese 


24 snare approves 


upenes merges on 
pian 


Approvals 

* Use pre-approved content where 
possible 

+ New content suggestions to GCS 
require a form with analysis, and 
proposed content ~ MGCS handles 
approvals and transiation 

«Errors found are managed urgently 
through web coordnator and changed 
rapidly 
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Overview — D&V Chatbot Dashboard 


Chatbot Dashboard has been built with collaboration and input from ITS, ServiceOntario and GSIC 


CONTACT 
CENTRE 
CHATBOT 
REPORT 
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What value does this report deliver? 


This report presents key performance indicators of the Drivers and Vehicles contact centre chatbot channels. Its goal is to 
enable executives and operational specialists to determine whether chatbots are making a positive contribution, achieving 
results, and aligned to our goals across the following Critical Success Factors: 


© Effectiveness: Metrics exhibiting how successfully the chatbot is achieving the desired results. 
FA Adoption: Metrics exhibiting how well the chatbot is being adopted by cusomers. 

z 

2% Satisfaction: Metrics indicating how satisfied chatbot customers are. 


What is the scope of this report? 


This report is based on the chatbot service for the Drivers and Vehicles contact centre. The sources include data from D&V 


webpages with a chatbot, customer conversations with chatbots, and call centre data for chatbot phone queues related to 
chatbot channels. 


Features in this report: Whom do I contact for support? 
Q Click on the magnifying glass to dive into the data To report an issue with this report, cick HERE 
used to calculate each metric 
To submit a general inquiry, click HERE 
@ Hover over the 'i' to view the logic used to calculate To request access to this report, click HERE 
each metric 


© Click on the back button to go back to the previous 
page 


+ The Chat window is part of the ECCS system 
- while the chats and content reside in the 
IBM Watson API 


+ The creation of this Dashboard has been 
led by ITS in partnership with GSIC 


+ Best practices have been reviewed and 
balanced with what can be measured in the 
systems that are involved 


+ The pilot is on the 5 webpages with a low 
or soft profile 


+ We are currently working to add online 


insights to each measure and ensuring 
dynamic updates are available. 


Ontario © 
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Dashboard Summary 


The summary offers a view to the volume flow from the webpage through to calls received at the Contact 


Centre 


aU aTAnalytics Contact Centre Chatbot Report - Summary (©) 


495K 


Total Pageviews 


4 


5.860 .. Etoo 


Chat Button Selections “Be | 


4 


3781 
Chats Q 


4 


5 6 5 The total number of calls made to the 
contact centre on the dedicated 


es chatbot queues that the bot provides. 


Effectiveness 
Metrics 


Adoption 
Metrics 


Satisfaction 
Metrics 
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This four-week period represents 20 business days during 
which the uptake of the chatbot was 1.18% of the webpage 
users 


Düring this period of resumption: 
Letters encouraging online renewals — millions. Confusion about G renewals, 
fines, months, scan code 
Emails sent about adding or opting out electronic renewals - 500K+ 
Issues with drive test appointment booking and volume of calls resulting in hang 
ups and chats and call to DVPI 
Boarders opened — increasing the need for the most current ID 


5,860 people selected the chat button and some* did not 
follow through on the chat — note they may still offer feedback 


Chats, like a contact centre calls are typically higher at the start 
of the week, system issue on the applications, changes to 
information, a long weekend 


* As with any minimum viable product used for pilot there are 
a few fixes to sort out — we are aware there are some dropped 
chats between IBM and ECCS. This is for future development. 


Ontario © 
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Dashboard Effectiveness 


The effectiveness demonstrates how well the questions are answered and the demand and willingness to call 


© (@) Chatbot Effectiveness 


= EEEN 2 * The Coverage rate moves between 80 to 95% — the chatbot is 
87.92% e 3 7 most effective closer to 80% and can be reset allowing more 
answers to be offered with a lower confidence 
e We know from the immediate feedback and daily audit of the 
chats that about 90% are getting the correct answer and do not 
indicate any issues with the chat 
* 50% of the chats happen outside of the contact centre hours — 
z 7 which shows there is demand for information and reassurance 
48.62% act Be ia sae beyond the open hours 
* The call abandonment rate is low: 
* Reflects new staffing 
A * Willingness to wait for the information 
cee o * Mondays are always higher than the remainder of the week 


Call Abandonment Rate 


bolas “Sabina © Ontario & 
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Dashboard Adoption 


The summary demonstrates how using and adopting the various elements of the chatbot experience 


© Chatbot Adoption ra] 


Uptake Rate ake Rale by Daie 


‘ll 
tl 


The uptake rate is steady between 1 to 1.5% on these 5 
pages and intentional low promotion for the pilot. There 
have been over 34,000 chats — adding volume to offer 
some perspective to data %. 


Gasber25 20 


The interaction rate helps us understand how many 
messages people are exchanging with the chatbot — some 
people have one and others are asking multiple things 


We consider the public self-serving a success and typically 
around 85% - (Chat Button Selections less the Calls divided 
by the Chat Button Selections) 


Sell Serve Aste 


90.36% 


This is the opposite of the calls received rate which is 
typically 15% - there is a dedicated number for chatbot 
callers to help with this measurement (no priority is 
currently given) 


1s 


141496 43.51% 


Ontario © 
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Dashboard Satisfaction 


The summary will improve with further iterations of the chatbot — offering people the option of feedback within 
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the chatbot and an understanding of what is measured 


Chatbot Satisfaction 


Immediate Customer Feedback by Date 


Immediate Customer 
Feedback 


100% 
i , 
È oso 97.70% 
89.08% i — 
£ 9305% oA 4 
= oo% | 82356 91.86% us \/ 
3 Vl V pom 
= 85% 
$ 86.10% 
A 85.90% Bu aiga 
E 
E so% 
Oct 24 Oct 31 Nov 07 Nov 14 


© Date 


November 8, 32 376 

2021 

November 15, 46 370 

2021 

November 16, 33 349 
1 

Total 640 5,860 


91.49% 


87.57% 
90.54% 


89.08%" 


Percentage Extremely Satisfied or Satisfied 


Percentage Extremely 
Satisfied or Satisfied 


@ percentage Exremaly Satisfied or satisfied @percentage Provides Ratings 


19.63% 


extremely satisfied 


October 20, 2021 7.8% 
October 22, 2027 195% 
November 5, 2021 149% 
October 26, 2021 12.9% 
November 4, 2021 17.6% 
October 31, 2021 7.8% 


November 2, 2021 


Total 


15 


Within the chatbot we ask if people received the answer 
they are looking for; yes or no 


89% of chats are leaving with the right answer and 
without saying no - very positive 


Analysis of the No’s tell us they do not like the policy; they 
need to come in, pay a fee, book a road test for G2, 
complex questions, no agent to call after hours 


Satisfaction - Top two box score on a 5 point Likert scale is 
the usual way to look at satisfaction and about 12% of the 
chats respond 


A rating and feedback can be left in ECCS - without ever 
using the IBM chatbot — i.e. rating it for being a “bot” 


Currently the satisfaction measures could use refinement 
and future versions of the chatbot will look at people 
being able to provide feedback throughout the experience 


The daily review of all chats helps the team understand 


what to improve and they share this with the business to 
influence positive change 


Ontario © 
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Thank you 


Kim Monastyrskyj 
Kim.Monastyrskyj@Ontario.ca 


